SPICE-0012: URL standard library by bioball · Pull Request #13 · apple/pkl-evolution

bioball · 2025-01-30T00:26:21Z

No description provided.

StefMa · 2025-01-30T06:06:31Z

It seems that "search params" are the correct technical wording. However I think if we would go with just Params it would be better understandable... 🤔 Maybe at least introducing an typealias for this? 🤔

Or are search params different to queries?

Just some random thoughts 🤷‍♂️ I've never seen SearchParams in Java/Kotlin yet (as an Android developer). Maybe others have more use for that wording...

stackoverflow

I Like the design. Left some questions in the comments.

spices/SPICE-0012-url-standard-library-module.adoc

stackoverflow · 2025-01-30T08:34:57Z

spices/SPICE-0012-url-standard-library-module.adoc

+/// The username component.
+///
+/// If the URL does not require a username, set to the empty string.
+username: AsciiString


What's the reason to make the empty string the "null" value instead of proper null like port, path, etc.?
I'd rather go with AsciiString?.

Good question. According to the spec, this field (and several other ones) do not admit null:

https://url.spec.whatwg.org/#url-representation

There are some fields that can be null, but this is not one of them.

Practically, I don't know if this makes any difference.

stackoverflow · 2025-01-30T08:35:19Z

spices/SPICE-0012-url-standard-library-module.adoc

+/// The password component.
+///
+/// If the URL does not require a password, set to the empty string.
+password: AsciiString


Same question as above.

spices/SPICE-0012-url-standard-library-module.adoc

stackoverflow · 2025-01-30T08:53:18Z

spices/SPICE-0012-url-standard-library-module.adoc

+  /// Parses [source] into a URL.
+  ///
+  /// Throws if [source] is an invalid URL.
+  external function parse(source: String): Url


I'm fine with having one version that throws on failure, for when users know their URL is correct. But we should have parseOrNull(source: String): Url? for when you want to recover from errors.

Our other parsers don't have a parseOrNull, so, this follows the same precedent (see yaml.Parser and json.Parser).

If you need to recover from errors, you can use test.catch

After chatting a little bit more on this, I'm convinced that we should just have a parseOrNull here, updated.

The rationale is: we also have similar "OrNull" methods in other places. It makes it much easier for users; e.g. Uri.parseOrNull(input) ?? Uri.parse("https://example.com")

We should have the YAML and JSON parsers conform to this as a future task.

stackoverflow · 2025-01-30T08:58:41Z

spices/SPICE-0012-url-standard-library-module.adoc

+  else null
+
+/// Creates a [SearchParams] from the given form encoded string.
+const function SearchParams(input: String): SearchParams = // etc


Is it possible input is not a valid query string? If so, what happens then?

Good question; I don't think so.

Here is the parsing algorithm: https://url.spec.whatwg.org/#urlencoded-parsing

None of these steps involve failure.

stackoverflow · 2025-01-30T09:06:58Z

spices/SPICE-0012-url-standard-library-module.adoc

+/// ```
+/// encode("https://example.com/some path/") == "https://example.com/some%20path"
+/// ```
+const external function encode(value: String): String


Suggested change

const external function encode(value: String): String

const external function encodeUri(value: String): String

encode is a bit too generic IMO, and it was hard to understand the difference between it and percentEncode at first glance.
Could also be called encodeForUri, encodeUriString, etc.

The module's name is already Url, so, usage looks like Url.encode. I think Url.encodeUri would be a little too wordy.

By the way, one of the suggestions of this SPICE is to prefer "URL" over "URI".

stackoverflow · 2025-01-30T09:14:30Z

spices/SPICE-0012-url-standard-library-module.adoc

+----
+import "pkl:Url"
+
+myUrl: Url = new { // <1>


parse will throw (or return null) on an invalid URL. What happens when you create an invalid URL with new? Will it throw?
We could have a UrlBuilder that allows you to create a URL with new and call toUrl(orNull) to validate it.

We can ensure that you cannot construct invalid Urls through property types and constraints.

bioball · 2025-01-30T15:31:51Z

@StefMa: this API is taken from the URL spec, see:

https://url.spec.whatwg.org/#urlsearchparams

By the way, Java's stdlib doesn't have a library for parsing the query. However, some other languages do, and call them different things:

Ruby: CGI
Dart: queryParameters
JavaScript: URLSearchParams
go: ParseQuery
Swift: queryItems

Technically, the query string portion of a URL is just a string (no structure). Representing structured data using & and = characters is URL form encoding.

KushalP · 2025-01-30T17:00:28Z

Documenting something here that we spoke about: will merging this into main require codegen implementations to exist as well?

HT154 · 2025-01-30T17:20:24Z

From an ergonomic standpoint, I would hope that Pkl's Url type produces language/stdlib-native URL types in each target language. Pretty sure would require changes in both the code generators and the Unmarshal/Decoder implementations.

bioball · 2025-01-31T17:13:56Z

Good point; I need to fill in some details about language bindings. And, I agree, it should probably turn into the target language's URL type.

HT154 · 2025-01-31T17:41:15Z

I wonder if it makes sense to expand read to accept Url values as well. Many of the places I'm using the experimental URI library today pass the "rendered" URI directly to read. This obviously isn't a huge impact, but might be a nice little bit of sugar.

Co-authored-by: Islon Scherer <islonscherer@gmail.com>

* Add `parseOrNull` to the parser * Add module-level `parse` and `parseOrNull` * Add section describing language bindings, and binary format * Add `value` property

bioball · 2025-01-31T18:56:01Z

I wonder if it makes sense to expand read to accept Url values as well. Many of the places I'm using the experimental URI library today pass the "rendered" URI directly to read. This obviously isn't a huge impact, but might be a nice little bit of sugar.

We possibly want to make read() also only accept string literals in the future (where exsiting dynamic reads would be replaced with a "run" mode of some sort). Without this, we can't statically analyze a module and determine all of its dependencies. So, with that in mind, I would rather keep read() in its current form.

HT154 · 2025-01-31T19:57:38Z

spices/SPICE-0012-url-standard-library-module.adoc

+.pkl.Url
+[source,pkl]
+----
+module pkl.Url


How would a URI like env:USER be represented by this module?

In go, this is parsed as

url.URL{ Scheme: "env", Opaque: "USER", }

This would be parsed as new URL { scheme = "env"; path = "USER" }.

See https://url.spec.whatwg.org/#example-url-components

Actually, I wonder how we can better support opaque URLs. The env URL env://foo/bar represents an env var with value //foo/bar, and doesn't mean "the host is foo and the path is /bar".

I'd like to proffer Go's URL implementation as a carrot and Swift's as the stick:

Go

Go's url.Parse function does (roughly) this:

Split off the fragment, if present, and decode/store it

Split off the scheme, if present

Split off the query, if present, and store it

If a scheme is present and the character immediately following the : is not /, store the remainder in Opaque and return

Otherwise, proceed to parse the authority and path

url.URL.String produces a string from a URL struct. When Opaque is non-empty, the result has form scheme:opaque?query#fragment, otherwise it's scheme://userinfo@host/path?query#fragment.

In practice, I've found this API to be extremely usable and flexible enough for working with the varied URIs found in the Pkl ecosystem. This may be a good design to reference that departs somewhat from the literal spec in the name of usability.

Swift

Having worked with pkl-swift, I've found Foundation URL type's lack of similar support makes working with opaque URIs like env:USER significantly less ergonomic. Parsing just the "USER" out requires error-prone mangling of the full URL string, eg. https://github.com/apple/pkl-swift/blob/c280a0e6324097c429bfc19a2cfe5761ab12bbe9/docs/modules/ROOT/pages/external-readers.adoc?plain=1#L33) to get the equivalent of Go's url.URL.Opaque.

Pkl

Given that opaque URIs are already commonplace in the Pkl ecosystem (and used for several of the runtime's built-in resources), I think having first-class support for them in this proposal is necessary. A similar model to Go's could be adopted, but with added type constraints so ensure that opaque is never non-null at the same time as hostname/port/path`.

A case like env://foo/bar (or env:/foo/bar) is still tricky (and go parses them "wrong"). I wonder if it would be reasonable to do something like this:

opaque: String = "<rendered authority/path>"

Url.toString() would then use this field and ignore the authority/path field. This would allow direct construction of the Url to directly set opaque while still maintaining the same behavior as non-opaque URLs. Similarly, during parsing, if an authority/path are found (same criteria as Go) then those fields are populated, otherwise opaque is set directly.

Thoughts?

You actually need to do wrangling too with Go's url API for opaque URLs. For example, to turn a URL like env://foo%20/bar into //foo /bar, you'd need:

func getSchemeSpecificPart(u *url.URL) (string, error) { return url.PathUnescape(strings.Split(u.String(), ":")[1]) }

Whether a URL is opaque or not is really up to the scheme, and a simple rule like "if the char after the colon is /, it's opaque" doesn't feel good enough.

Add SPICE for URL standard library

9709094

bioball force-pushed the url-spice branch from 7eddd19 to 9709094 Compare January 30, 2025 00:31

bioball mentioned this pull request Jan 30, 2025

URI standard library apple/pkl#911

Open

Add searchParams property

fc2b167

stackoverflow reviewed Jan 30, 2025

View reviewed changes

bioball and others added 3 commits January 31, 2025 09:49

Update spices/SPICE-0012-url-standard-library-module.adoc

bb785df

Co-authored-by: Islon Scherer <islonscherer@gmail.com>

Update spices/SPICE-0012-url-standard-library-module.adoc

a214ae4

Co-authored-by: Islon Scherer <islonscherer@gmail.com>

Updates

e83369d

* Add `parseOrNull` to the parser * Add module-level `parse` and `parseOrNull` * Add section describing language bindings, and binary format * Add `value` property

HT154 reviewed Jan 31, 2025

View reviewed changes

	const external function encode(value: String): String
	const external function encodeUri(value: String): String

Conversation

bioball commented Jan 30, 2025

Uh oh!

StefMa commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stackoverflow left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bioball Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bioball Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bioball Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bioball commented Jan 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KushalP commented Jan 30, 2025

Uh oh!

HT154 commented Jan 30, 2025

Uh oh!

bioball commented Jan 31, 2025

Uh oh!

HT154 commented Jan 31, 2025

Uh oh!

bioball commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bioball Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Go

Swift

Pkl

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

StefMa commented Jan 30, 2025 •

edited

Loading

bioball Jan 30, 2025 •

edited

Loading

bioball Jan 31, 2025 •

edited

Loading

bioball Jan 30, 2025 •

edited

Loading

bioball commented Jan 30, 2025 •

edited

Loading

bioball commented Jan 31, 2025 •

edited

Loading

bioball Jan 31, 2025 •

edited

Loading